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ABSTRACT 


This  study  investigated  the  impact  of  lowering  the  minimum  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB)  Mechanical  composite  (MECH)  scores  required  for  recruits  to  enter 
maintenance  career  fields.  The  sample  included  (N=48,009)  Air  Force  technical  school  trainees 
who  attended  school  between  1990  and  1995.  A  contingency  table  showing  the  relationship 
between  predictor  scores  and  success/failure  in  technical  school  and  logistic  regression  analyses 
suggested  that  required  scores  should  be  raised  for  five  Air  For  ce  Specialties  (AFSs)  and  should 
remain  at  the  present  level  for  four  others.  No  linear  relationship  was  apparent  between  test 
scores  and  technical  school  grades  or  pass/fail  criteria  for  two  AFSs.  Results  provided  little 
evidence  that  reducing  minimum  MECH  score  requirements  slightly  will  increase  the  rate  of 
technical  school  failures.  The  need  to  collect  technical  school  grades  for  unsuccessful  trainees 
was  identified. 
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The  Effects  of  Lowering  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) 
Mechanical  Score  Requirements  on  the  Number  of  Applicants  Eligible  for  Training 
in  Maintenance  Occupations  and  the  Percentage  of  Training  Failures 

Maintenance  occupations  play  a  critical  role  in  sustaining  the  Air  Force's  mission 
capabilities  in  combat  operations  and  strategic  airlift.  It  has  become  increasingly  difficult 
to  sign  up  enough  high  quality  recruits  to  fill  Air  Force  entry-level  jobs  in  aircraft,  vehicle, 
and  equipment  maintenance  (Chapman,  1996).  If  present  trends  continue,  there  may  come 
a  time  when  recruiters  will  be  unable  to  fill  manpower  requirements  for  these  Air  Force 
Specialties  (AFSs)  and  the  service's  ability  to  accomplish  its  mission  will  be  constrained  by 
the  lack  of  qualified  technicians.  This  report  investigates  one  approach  toward  solving  this 
problem. 

Cut-off  scores 

One  way  to  increase  the  number  of  applicants  eligible  for  AFSs  involving  mechanical 
skills  (referred  to  as  maintenance  occupations  in  the  rest  of  the  paper)  is  to  lower  the 
minimum  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  mechanical  aptitude 
(MECH)  scores  required  for  entry  into  these  jobs.  Cut-off  scores  have  been  established  for 
each  AFS  to  ensure  recruits  entering  initial  technical  training  have  at  least  the  minimum 
level  of  ability  needed  to  complete  the  training  successfully.  If  cut-off  scores  are  set  too 
low,  technical  training  failure  rates  will  increase,  leading  to  higher  recruiting  and  training 
costs.  On  the  other  hand,  if  cut-off  scores  are  set  too  high,  applicants  who  would  be 
successful  in  technical  school  will  be  erroneously  rejected.  Score  requirements  that  are 
higher  than  necessary  reduce  the  size  of  the  pool  of  applicants  eligible  for  a  job,  make 
recruiting  more  difficult,  and  increase  the  likelihood  that  minority  applicants  will  be 
excluded.  MECH  score  requirements  are  a  tradeoff  between  the  competing  needs  to  keep 
manning  at  acceptable  levels  and  to  obtain  recruits  capable  of  completing  technical  school 
training  on  very  complex  systems.  Before  making  any  changes,  it  is  necessary  to  estimate 
the  impact  of  changes  on  the  number  of  applicants  eligible  for  technical  training  and  on 
academic  failure  rates  expected  for  various  levels  of  aptitude  scores. 

Changes  in  training  criteria 

The  technical  training  criterion  scores  available  for  this  study  span  a  period  marked  by 
change.  It  includes  Desert  Storm/Desert  Shield,  implementation  of  a  new  wing  structure, 
the  relocation  of  several  technical  training  schools  from  bases  being  closed  to  other  sites, 
and  a  continuing  effort  to  downsize  the  force.  Utilization  and  Training  Workshops  in 
many  AFSs  also  directed  changes  in  course  content  and  length  that  directly  affect  training 
criteria. 

Another  significant  change  is  the  growing  use  of  work  sample  tests  in  technical  schools. 
Instructors  in  courses  that  train  recruits  to  the  level  necessary  for  Mission  Ready 
Technician  (MRT)  certification  must  grade  trainees'  performance  on  a  large  number  of 
critical  tasks.  If  much  weight  is  given  to  these  work  sample  performance  assessments  in 
technical  training,  it  is  necessary  to  demonstrate  the  validity  and  reliability  of  these 
measures  (American  Psychological  Association,  1985).  They  must  also  be  incorporated  in 
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the  selection  criteria  used  in  validation  research.  Work  sample  criterion  measures  were  not 
available  for  the  present  study. 

Previous  research 

Previous  studies  have  examined  the  validity  of  ASVAB  composites  for  predicting 
technical  school  grades  over  a  large  range  of  jobs  (e.g.,  Wilboum,  Valentine,  &  Ree,  1984) 
or  in  examining  the  potential  to  improve  prediction  by  combining  ASVAB  sub-tests  in 
different  ways  (e.g.,  Ree  &  Earles,  1992).  There  has  been  little  research  addressing  the 
impact  of  different  cut-off  scores  on  the  number  of  applicants  eligible  for  specific 
occupational  specialties,  or  on  the  effect  of  lower  minimum  qualifying  scores  on  academic 
failure  rates.  An  important  goal  of  this  study  was  to  determine  whether  or  not  cut-off 
scores  could  be  lowered  to  increase  the  number  of  applicants  eligible  for  maintenance  jobs 
without  increasing  technical  school  failure  rates  to  unacceptable  levels.  For  the  purposes 
of  this  study  failure  rates  up  to  and  including  five  percent  were  deemed  to  be  acceptable. 

This  study’s  contribution 

Analyses  for  each  AFS  were  expected  to  support  one  of  the  following  conclusions: 

1 .  There  is  no  evidence  of  a  linear  relationship  between  MECH  scores  and  criteria 
representing  technical  training  success.  Additional  research  is  needed  to  develop 
and  validate  predictors  for  the  AFS. 

2.  There  is  evidence  that  a  linear  relationship  exists  between  MECH  scores  and  criteria 
representing  technical  training  success,  and  . . . 

a.  The  MECH  cut-off  score  for  an  AFS  should  be  raised  because  the  failure  rate 
associated  with  the  current  MECH  cut-off  score  is  unacceptably  high. 

b.  The  MECH  cut-off  score  for  an  AFS  should  not  be  lowered  because  analyses 
predict  an  unacceptably  high  failure  rate  for  applicants  with  MECH  scores  below 
the  current  cut-off  score. 

c.  The  MECH  cut-off  score  for  an  AFS  can  be  lowered  incrementally  because  the 
rate  of  failures  is  very  low  and  there  is  no  evidence  that  failure  rates  would  increase 
to  unacceptable  levels  if  the  cut-off  scores  were  lowered  a  few  points. 

This  approach  acknowledges  that  decisions  based  on  predictor-criterion  information 
depend  in  part  on  the  type  and  quality  of  information  available.  It  also  recognizes  that 
decisions  about  the  allocation  of  scarce  resources  (in  this  case  applicants  with  high  MECH 
scores)  must  consider  a  variety  of  issues  that  go  beyond  the  kind  of  data  analysis  reported 
here.  This  paper  is  intended  to  support  the  decision  process. 


OBJECTIVES 

The  objectives  of  this  study  were:  (1)  to  investigate  the  quality  of  recruits  entering 
maintenance  technical  training  courses  in  recent  years,  (2)  to  examine  the  validity  of  the 
predictors  of  success  in  technical  training  presently  in  use,  and  (3)  to  estimate  the  effect  of 
lowering  ASVAB  mechanical  test  scores  on  technical  school  failure  rates  and  the  number 
of  applicants  expected  to  qualify  for  entrance  into  maintenance  AFSs. 

METHOD 

Subjects 

Subjects  were  48,009  first  term  recruits  who  entered  the  service  between  January,  1990 
and  September,  1995,  and  were  assigned  to  a  mechanical  AFS.  There  were  42,980 
graduates,  1,229  academic  failures,  and  1,050  subjects  who  were  eliminated  because  of 
medical  problems  or  other  non-academic  reasons.  The  remaining  2,750  subjects  either  did 
not  complete  Basic  Military  Training  or  were  re-routed  to  a  non-mechanical  technical 
school.  Demographics  are  shown  in  Table  1. 


TABLE  1.  CHARACTERISTICS  OF  SUBJECTS  IN  THE  STUDY 


PERCENTAGE  OF  ENLISTEES  BY  YEAR  OF  ENLISTMENT 


YEAR 

m— 

Iffl  ; 

IliPiil 

— W 

g!H4 

■H 

— 

Number: 

865 1 

8173 

10503 

7384 

8640 

4658 

48009 

1 H 


1 8  or  Less 

24,2 

21.2 

20.6 

25.2 

21.1 

11.6 

2i.3 

19  or  20 

51.4 

47.0 

48.7 

46.7 

46.7 

49.5 

48.7 

21  or  More 

24.4 

31.8 

30.7 

28.1 

28.1 

38.9 

30.0 

ill 


MALE 

FEMALE 

94.7 

05.3 

94.0 

06.0 

94.8 

05.2 

95.3 

04.7 

95.6 

04.4 

95.8 

04.2 

95.0 

05.0 

^JLACE 

WHITE 

89.1 

90.3 

90.4 

87.5 

86.8 

84.2 

88.5 

BLACK 

08.1 

06.4 

06.7 

08.7 

08.0 

08.6 

07.6 

OTHER 

02.7 

03.2 

02.9 

03.8 

05.2 

07.3 

03.9 

Note:  Figures  for  1995  are  based  on  partial  year  data. 


Predictors 

Subjects'  scores  on  the  ASVAB  Mechanical  (MECH)  composite  were  used  as  the  main 
predictor.  The  MECH  score  is  the  sum  of  the  Mechanical  Comprehension  (MC),  General 
Science  (GS),  and  Auto  and  Shop  Information  (AS)  subtests  of  the  ASVAB.  Ree  and 
Earles  (1992)  reported  the  internal  consistency  reliability  for  the  MECH  composite  was  .90 
(N=88,724).  Entrance  requirements  for  some  AFSs  specified  minimum  MECH  scores  and 
one  other  ASVAB  composite  score  (i.e.,  electrical,  general,  or  administrative).  In  these 
cases,  MECH  was  used  as  the  predictor  by  itself,  and  in  combination  with  the  other  score. 
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Criteria 

There  were  two  criterion  measures.  Final  School  Grades  (FSGs)  were  used  in 
correlational  analyses.  FSGs  were  computed  as  the  average  of  the  subjects’  scores  on 
written  tests  they  completed  during  technical  training.  Pearlman,  Schmidt,  and  Hunter 
(1980)  reported  the  alpha  reliability  of  FSGs  as  .80.  A  limitation  of  the  current  study  is 
that  FSG  scores  were  available  only  for  subjects  who  had  successfully  completed  training. 
The  lack  of  criterion  measures  covering  the  full  range  of  subject's  scores  has  been  noted  in 
other  research  for  the  Air  Force  (Ree  &  Earles,  1992)  and  Navy  (Borack,  1996).  FSG 
scores  in  the  current  study  ranged  from  the  minimum  passing  score  of  70  to  99.  This 
restriction  in  range  directly  impinges  on  the  variability  of  the  criterion  exacerbating  the 
range  restriction  that  occurs  as  a  consequence  of  explicit  selection  on  the  predictor  (MECH 
scores).  A  pass/fail  criterion  measure  was  used  in  logistic  regression  analyses.  It  was 
available  for  students  who  were  successful  and  who  failed  academically.  Those  who  did 
not  complete  training  for  non-academic  reasons  were  excluded. 

Criterion  groups 

The  use  of  cut-off  scores  is  based  on  the  assumption  that  the  level  of  aptitude  required 
for  success  in  training  varies  among  the  technical  training  courses.  Thus,  different  cut-off 
scores  are  set  for  different  AFSs.  Previous  validation  studies  (e.g.,  Wilboum,  et  al.,  1984) 
have  implicitly  assumed  that  individual  courses  and  associated  criterion  measures  are  fairly 
consistent  over  time.  In  view  of  all  the  changes  in  course  content,  length,  location,  and 
emphasis  that  had  occurred  over  the  five  years  of  data  available  for  the  present  study,  it 
was  clear  that  course  criteria  had  changed  for  at  least  some  AFSs.  Thus  it  was  necessary  to 
avoid  combining  groups  of  recruits  who  were  trained  in  the  same  AFS  at  different  times 
and  whose  courses  and  course  criterion  measures  had  differed  substantially.  The  method 
outlined  below  was  used  to  decide  which  training  groups  for  a  single  AFS  were 
homogenous  enough  to  be  combined  and  which  groups  differed  in  some  way. 

1 .  Training  course  managers  for  each  AFS  provided  information  on  changes  in  course 
content,  emphasis,  length,  and  location  that  might  cause  differences  in  the  criteria.  As 
a  result  1-4  criterion  sub-groups  were  established  for  each  AFS  included  in  the  study. 

2.  Procedures  described  by  Cohen  (1988)  were  used  to  test  the  differences  between 
correlations  of  MECH  and  FSG  scores  for  each  pair  of  training  groups  in  the  same 
AFS.  Since  MECH  scores  were  stable  between  1990  and  1995,  finding  a  significant 
difference  in  two  correlations  seemed  likely  to  reflect  changes  in  FSG  criteria. 

a.  If  differences  in  the  correlations  were  not  large  enough  to  be  significant,  the 
training  groups  were  combined  in  subsequent  analyses. 

b.  If  the  differences  were  large  enough  to  be  significant,  the  training  groups  were 
kept  separate. 

Additional  information  on  the  statistical  test  and  the  correlations  between  MECH  and  FSG 
scores  for  the  initial  groups  can  be  found  in  Appendix  A. 
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RESULTS 


Quality  of  accessions 

Table  2  shows  that  the  quality  of  recruits  entering  maintenance  occupations  between 
January,  1990  and  September,  1995  was  high.  Nearly  all  were  high  school  graduates  who 
had  earned  MECH  scores  that  placed  them  in  the  top  three  enlistment  categories. 


TABLE  2.  QUALITY  OF  ACCESSIONS  BY  YEAR  OF  ENLISTMENT 

ASVAB  MECHANICAL  APTITUDE  SCORES 


1  1990  1 

mmm 

mmm 

Ilfli 

illiil 

■HI 

TOTAL 

MEAN 

72.5 

72.7 

72.6 

71.3 

72.3 

71.6 

72.3 

ST  DEV 

14.2 

14.4 

14.3 

14.8 

13.8 

14.2 

14.3 

N 

8591 

8112 

10401 

7137 

8190 

4363 

46794 

PERCENTAGE  OF  RECRUITS  IN  EACH  ENLISTMENT  CATEGORY 


CATEGORY  H|| 

:•  T9J>r 

1IHII 

iliiii 

mm  ii 

mmm 

mmm 

TOTAL 

I  (Highest) 

01.9 

02.7 

02.9 

02.7 

04.2 

04.4 

03.0 

II 

38.1 

42.0 

40.4 

38.9 

45.7 

43.0 

41.2 

III 

59.6 

55.1 

56.2 

58.2 

49.9 

52.2 

55.4 

IV  (Lowest) 

00.3 

00.0 

00.4 

00.1 

00.3 

00.4 

00.3 

TOTALS 

8591 

8112 

10401 

7137 

8190 

4363 

46794 

PERCENTAGE  OF  RECRUITS  WITH  HIGH  SCHOOL  DIPLOMAS 


EDUCATION  3|||: 

■SHI 

Hi 

mimm 

an 

WfflM 

wmm 

mmm 

HS  DIPLOMA 

98.8 

99.1 

98.6 

98.8 

98.5 

98.1 

98.7 

GED 

1.0 

0.8 

1.3 

1.1 

1.4 

1.7 

1.2 

NO  DEGREE 

02 

QJ. 

QJ. 

QJ. 

QJ. 

QJ 

QJ. 

TOTALS 

8651 

8173 

10503 

7384 

8640 

4658 

48009 

PERCENTAGE  OF  RECRUITS  SUCCESSFUL  IN  TECHNICAL  TRAINING 


OUTCOMES  %& 

1990 

mmm 

11*1! 

ill 

mmm 

mmm 

iliiii! 

SUCCESS 

89. 1 

88.2 

89.4 

93.7 

94.5 

mi 

42980 

FAILURE 

3.2 

2.8 

3.4 

2.0 

1.7 

1.5 

1229 

OTHER 

13. 

M 

12 

42 

13 

212 

3800 

TOTALS 

8651 

8173 

10503 

7384 

8640 

4658 

48009 

NOTE:  1995  data  is  incomplete.  Recruits  who  had  entered  training  in  1995,  but  had  not  completed 
it  at  the  time  the  data  were  collected  are  included  in  the  "other"  category. 


Training  completion  data  for  1990-1994  show  that  the  proportion  of  trainees  who 
successfully  completed  technical  training  increased  slightly  during  this  period. 


Table  3  shows  that  the  quality  of  the  recruits  entering  each  AFSs  remained  consistently 
high  between  1990  and  1995.  Two  rows  are  shown  for  the  same  AFS  in  cases  where  two 
groups  of  students  experienced  different  training  conditions  or  grading  standards  because 
of  changes  in  training  content,  methods,  or  objectives. 
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TABLE  3.  MEAN  ASVAB  MECHANICAL  APTITUDE  (MECH)  SCORES  FOR  RECRUITS  ENTERING 

MAINTENANCE  TECHNICAL  TRAINING  (1990-1995) 


(N 

p 

GO 

CO 

GO 

CO 

CN 

in 

in 

CN 

vo 

H 

CN 

p 

P 

CN 

in 

vo 

CO 

P 

vo 

CN 

O 

CO 

p 

J5 

p 

H 

O 

p 

VO 

in 

CN 

in 

CO 

O 

vo 

00 

p 

o 

O 

p 

p 

rH 

P 

00 

p 

P 

CN 

m 

CO 

in 

P 

O 

p 

CN 

CN 

in 

(N 

CN 

H 

rH 

rH 

rH 

in 

CO 

vo 

Ol 

CO 

o 

CO 

in 

P 

CO 

vo 

CO 

H 

<N 

H 

CN 

H 

H 

rH 

P 

VO 

H 

P 

in 

p 

vo 

H 

00 

ai 

p 

p 

Ol 

p 

CO 

CN 

P 

CO 

in 

P 

CN 

CO 

CN 

H 

ft£  _  5} 

10 

CN 

P 

VO 

P 

o 

o 

H 

CO 

P 

CO 

01 

p 

CN 

CN 

vo 

CO 

P 

rp 

o 

CN 

O 

VO 

VO 

H 

g  j  i 

P 

rH 

CO 

CO 

CN 

p 

CN 

P 

rH 

CO 

CO 

p 

o 

H 

P 

CO 

LO 

P 

P 

vo 

p 

H 

CO 

CN 

VO 
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p 

p 

P 

p 
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p 
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P 
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P 

P 

P 

p 

p 

P 

P 

P 
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CO 

o 

o 
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o 

00 

p 
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CN 

01 
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H 

rH 

CO 

01 

01 
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p 

CN 

VO 

VO 
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CO 

in 

P 

CN 

H 

o 

CN 

00 
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CO 
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in 

CO 

01 
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VO 

CO 

o 

CO 

VO 

CN 

vo 

CO 

p 

CO 

ai 

CO 

p 

CN 

CO 

Q1 

m 

CO 

CN 

CO 

CN 
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p 

p 

p 

p 

p 

p 

p 

P 

p 

p 

p 

VO 

p 

P 

p 

VO 

p 

p 

p 

P 

m 

CO 

p 

CN 

VO 

p 

p 

in 

01 

CO 

Ol 

vo 

P 

rH 

m 

00 

CN 

vo 

H 

in 

H 

00 

CO 

p 

P  55 

O*  < 

00 

o 

CO 

CN 

CN 

CO 

00 

vo 

Ol 

vo 

p 

o 

VO 

H 

p 

VO 

p 

Ol 

p 

00 

in 

CN 

P 

Cl  fid 

p 

oi 

CN 

CO 

CO 

CN 

h 

CO 

O 

CN 

CO 

vo 

01 

Ol 

01 

CN 

01 

CN 

in 

H 

CN 

CN 

CO 

H  S 

p 

VO 

p 

p 

p 

p 

p 

p 

P 

P 

p 

p 

P 

VO 

p 

P 

p 

P 

p 

P 

p 

P 

vo 

ro  a 
a*  < 

CO 

CO 

in 

VO 

CN 

00 

ci 

O 

in 

in 

CO 

CN 

CO 

o 

CO 

01 

H 

P 

vo 

P 

H 

p 

CO 

p 

o 

CO 

p 

p 

in 

in 

CO 

CO 

H 

CO 

p 

P 

co 

CO 

CN 

p 

P 

00 

_ ai  w 

p 

CO 

H 

CN 

P 

H 

CO 

rH 

P 

CO 

in 

H 

00 

H 

P 

00 

00 

P 

p 

o 

CO 

tH  jS 

p 

p 

CO 

P 

P 

p 

p 

P 

p 

CO 

vo 

P 

VO 

CO 

P 

p 

p 

VO 

vo 

p 

vo 

« 

Q 

a  o 
«  u 

CQ 


BSi 


£ 


►3  W 

o  w 

§  S 
8  £ 
M 

W 

S3 
w  a 

H  U 


ai 

CN 

VO 

CN 

H 

01 

CN 

o 

-p 

p 

o 

p 

o 

ij)  in 

ai 

VO 

00 

o 

in 

TP 

o 

in 

in 

o 

co 

p 

a 

CO 

CO 

^P 

CN 

00 

00 

<P 

p 

m 

p 

CN 

m 

0 

A 

p 

P 

P 

VO 

P 

p 

p 

p 

p 

p 

p 

p 

vo 

ft 

0) 

- 

ro 

H 

P 

01 

in 

CO 

rH 

CN 

o 

00 

o 

rH 

vo 

4-> 

4-> 

(H 

CO 

H 

CO 

CN 

^p 

CO 

P 

VO 

co 

H 

Ol 

rH 

p 

u 

M 

Ctf 

vo 

in 

rH 

o 

rH 

CO 

01 

CN 

vo 

CN 

P 

cn 

a 

u 

p 

P 

p 

P 

p 

00 

P 

P 

p 

p 

P 

P 

vo 

4H 

u 

•rH 

ai 

<J\ 

00 

CO 

H 

01 

VO 

rH 

vo 

p 

01 

O 

CN 

ctf 

a) 

G 

Cn 

CO 

CO 

o 

'P 

CN 

CO 

CN 

CO 

01 

o 

in 

CO 

p 

•H 

ro 

CO 

CN 

in 

H 

Ol 

P 

Ol 

p 

vo 

vo 

CN 

00 

G 

05 

p 

p 

p 

p 

P 

p 

P 

p 

p 

p 

p 

P 

vo 

T5 

0 

<D 

c 

TJ 

II 

rH 

rH 

rH 

rH 

rH 

H 

H 

H 

H 

rH 

rH 

rH 

rH 

sp 

^p 

p 

H 

p 

H 

p 

p 

in 

rH 

rH 

p 

3 

0} 

in 

in 

in 

in 

in 

in 

in 

in 

in 

LO 

in 

in 

in 

<P 

'p 

in 

in 

in 

in 

in 

in 

p 

in 

LO 

p 

o 

C 

n 

•H 

TJ 

CD 

0) 

4J 

vo 

CO 

vo 

01 

in 

vo 

co 

vo 

*p 

VO 

01 

in 

01 

H 

p 

H 

00 

H 

co 

CN 

co 

co 

CN 

VO 

CO 

u 

Ctf 

ro 

in 

CO 

ro 

CO 

in 

^P 

^P 

^P 

CO 

in 

^p 

CN 

CN 

CO 

^P 

co 

CN 

ro 

H 

co 

CN 

p 

CN 

rH 

ctf 

U 

w 

*H 

TJ 

a) 

a 

w 

*H 

01 

01 

CTi 

oi 

01 

01 

Ol 

Ol 

01 

Ol 

01 

Ol 

Ol 

CO 

Ol 

01 

01 

Ol 

01 

Ol 

Ol 

01 

01 

Ol 

Ol 

m 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

ctf 

cu 

p 

in 

in 

m 

in 

m 

m 

m 

m 

in 

in 

m 

CO 

LO 

LO 

m 

in 

in 

CN 

in 

in 

in 

in 

in 

H 

0) 

01 

ai 

CTi 

Ol 

Ol 

01 

Ol 

01 

Ol 

Ol 

01 

01 

01 

01 

01 

01 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

01 

01 

01 

U 

-H 

>> 

vo 

CN 

CO 

VO 

CN 

vo 

01 

<A 

CN 

CN 

vo 

01 

01 

vo 

Ol 

CO 

co 

vo 

vo 

VO 

vo 

co 

vo 

CO 

vo 

XI 

•> 

U 

o 

H 

O 

o 

H 

o 

o 

o 

H 

H 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

u 

a) 

o 

^P 

o 

o 

O 

CO 

co 

ro 

CO 

ro 

*p 

'p 

co 

o 

CO 

o 

o 

o 

o 

o 

co 

p 

o 

o 

o 

-H 

XJ 

Ol 

CTi 

CTi 

Ol 

01 

Ol 

01 

Ol 

01 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

Ol 

01 

01 

Ol 

4J 

0 

W  W  w  <  <  PQ 

-  oj 


<I<CQOW<PQOQ 

oococoroooHrHrHHrHr-iTHCNr-irHrHCNcopininvoHro 

cococococoinintninininininvovoioioinioiovovop^ 

<<<<<<<<<<<<<<<<<<<<<<< 


0)  W 
jj  m 
m  a) 

u  rH 


in 

o 


H 
TJ 
C 
H 

05 
a) 

4J  v 

«S  a 

TJ  ^ 

Cn  x> 
£  C 

H  ftf 

a  u 

H  -H 
ftf  4-1 
*H  *H 

h  a 
tn 

H 

..  CQ 

® 

jj  a) 


vo 


TABLE  3.  MEAN  ASVAB  MECHANICAL  APTITUDE  (MECH)  SCORES  FOR  RECRUITS  ENTERING 

MAINTENANCE  TECHNICAL  TRAINING  (1990-1995) 
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The  usefulness  of  MECH  scores  for  predicting  training  outcomes 

Correlations  between  MECH  scores  and  FSG  are  shown  in  the  third  column  of  Table  3. 
The  average  correlation  across  all  AFS  groups  was  r=.28  (N=39246).  The  pattern  of 
correlations  suggests  that  trainees  with  low  MECH  scores  would  fail  more  often  than  those 
with  medium  or  high  scores. 

MECH  scores  and  the  number  of  qualified  applicants 

Figure  1  compares  the  distribution  of  scores  for  high  school  students  who  expressed 
interest  in  the  Air  Force  with  the  distribution  of  scores  for  those  who  actually  joined  the 
Air  Force.  It  shows  the  percentage  of  individuals  scoring  at  each  level  in  the  applicant  and 
recruit  samples  multiplied  by  the  number  of  persons  in  those  groups  in  an  average  year. 


It  is  clear  that  recruits  entering  maintenance  training  had  average  to  high  MECH  scores, 
and  that  the  availability  of  potential  trainees  varies  across  MECH  score  ranges. 

MECH  scores  and  success/failure  in  training 

Finding  that  the  number  of  failures  increases  as  MECH  scores  decrease  would  be 
consistent  with  the  hypothesis  that  the  MECH  score  is  an  important  predictor  of  success  in 
technical  school.  A  contingency  table  was  developed  to  examine  the  relationship  between 
the  level  of  MECH  scores  and  the  number  of  failures.  Table  4  provides  little  support  for 
this  point  of  view.  Failures  occurred  at  nearly  all  levels  of  MECH  scores.  On  the  other 
hand,  MECH  scores  do  predict  FSG  scores  for  trainees  who  are  successful  in  training  in 
most  AFSs.  This  suggests  that,  given  an  adequate  level  of  ability,  slight  differences  in 
ability  among  trainees’  do  not  predict  success  or  failure. 

Table  4  also  shows  that  only  a  few  current  training  courses  have  failure  rates  greater 
than  or  equal  to  5  percent  for  any  range  of  MECH  scores.  The  applicant  sample  was  used 
to  estimate  the  percentage  of  the  applicant  population  with  scores  less  than  or  equal  to  the 
lowest  score  in  each  range  of  scores.  These  estimates  are  shown  at  the  bottom  of  Table  4. 
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TABLE  4.  CONTINGENCY  TABLE:  CUMULATIVE  PERCENTAGE  FAILING  FOR  VARIOUS  MECH  SCORES 
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[Percentage  of  Applicants  with  Lower  Scores _ 97.0  92.6  87.3  81.6  75.9  68.2  61.5  54.5  48.5  42.2  36.3  29.3  23.7 

Note:  Table  entries  show  the  cumulative  percentage  of  academic  failures  for  (1990-1995)  trainees  as  MECH  scores  decrease.  Scores  in  the  shaded  area  do  not 
meet  current  standards  for  entry  into  the  AFS.  Failure  rates  of  5%  or  higher  are  shown  in  bold  print. 


TABLE  4.  CONTINGENCY  TABLE:  CUMULATIVE  PERCENTAGE  FAILING  FOR  VARIOUS  MECH  SCORES 


not  meet  current  standards  for  entry  into  the  AFS.  Failure  rates  of  5%  or  higher  are  shown  in  bold  print. 


For  the  three  years  of  data  included  in  that  sample,  there  were  about  40,000  applicants 
each  year,  so  a  one  percent  change  would  represent  about  400  applicants  per  year.  The 
bottom  row  of  Table  4  shows  the  percentage  of  applicants  with  lower  scores.  The  impact 
of  lowering  cut-off  scores  from  55  to  45  can  be  estimated  by  using  the  percentages  on  the 
bottom  row  of  the  appropriate  columns  to  calculate  the  number  of  additional  applicants  that 
would  be  eligible  at  the  lower  cut-off  score.  In  this  case  (48.5-42.2)=6.3  and  6.3  x  400  = 
2520,  so  about  2520  more  applicants  would  qualify.  Note  that  the  number  of  applicants 
differs  across  the  range  of  MECH  scores.  Appendix  B  contains  more  accurate  information. 

Linear  regression 

The  empirical  information  provided  in  Table  4  can  be  supplemented  with  other  validity 
information.  Procedures  typically  involve  developing  a  linear  regression  model  in  the  form 
of:  FSG  =  B0+  B](MECHl )  and  substituting  values  of  the  independent  variable  MECH  into 
the  equation  to  compute  a  predicted  value  for  the  dependent  variable,  FSG  (Cascio,  1987). 
In  the  equation,  B0  represents  the  Y-intercept  and  B,  represents  the  increase  in  FSG  for  a 
one  unit  increase  in  the  MECH  score.  Unfortunately,  41  of  the  47  Y-intercept  ( B0 )  values 
computed  for  the  current  data  exceeded  the  passing  FSG  score  level  of  70.0.  Since  there 
are  no  MECH  scores  below  1,  adding  B,  (MECH)  to  B0  will  increase  the  predicted  score 
farther  above  the  failing  score  for  these  cases.  Thus,  the  lack  of  data  for  the  full  range  of 
criterion  scores  (i.e.,  FSG  scores  for  unsuccessful  as  well  as  the  successful  students)  was  a 
serious  impediment  to  this  analysis. 


Logistic  regression 

Logistic  Regression  analyses  using  the  pass/fail  criterion  were  completed  to  supplement 
the  contingency  table  information  in  Table  4.  The  usefulness  of  MECH  scores  in 
predicting  success  or  failure  was  evaluated  by  examining  the  significance  of  path 
coefficients,  determining  whether  or  not  the  model  predicted  the  pass/fail  outcomes,  and  by 
testing  the  model’s  ability  to  explain  the  variance  in  the  data  (Hosmer  &  Lemeshow, 

1989).  The  odds  of  failing  to  were  calculated  for  each  set  of  five  MECH  scores  ranging 
from  30  to  99  for  each  current  AFS  using  equations  (1  and  2)  from  by  Fox  (1984). 


M*)= 


l 


l  +  e~2 

where:  z  =  -(a  +  Bxi) 
and:  es  2.718 


(1) 


odds  = 


E(z) 
1-A  (z) 


(2) 


Results  (Table  5)  show  that  the  odds  of  failing  at  different  MECH  score  levels  differ 
among  criterion  groups.  Given  the  goal  of  maintaining  a  failure  rate  under  5%,  Table  5 
suggests  cut-off  scores  should  be  raised  for  five  AFSs  and  lowered  for  four  others. 
Although  the  logistic  regression  models  for  these  AFSs  fit  acceptably  well,  none  of  them 
predicted  any  failures.  The  logistic  regression  models  obtained  for  the  other  criterion 
groups  did  not  fit  the  data  well.  Detailed  results  can  be  found  in  Appendix  C. 
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TABLE  5.  CUMULATIVE  PERCENTAGE  OF  TRAINEES  EXPECTED  TO  FAIL  FOR  VARIOUS  MECH  SCORES 
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Analyses  for  AFSs  requiring  minimum  scores  on  the  MECH  and  one  other  ASVAB 
aptitude  test,  or  AFSs  in  which  qualification  required  a  minimum  score  on  either  the 
MECH  or  another  ASVAB  composite  are  shown  in  Table  5.  Acceptable  fit  statistics  and 
significant  beta  weights  for  MECH  and  ASVAB  electrical  aptitude  test  (ELEC)  scores 
were  obtained  for  two  AFSs:  2A6X2  (Aerospace  Ground  Equipment  Maintenance)  and 
3E1X1  (Heating,  Ventilation  and  Air  Conditioning).  In  both  cases  adding  ASVAB  ELEC 
to  the  model  improved  prediction  significantly  (p<.05).  Other  logistic  regression  analyses 
provide  evidence  that  ELEC  may  be  useful  in  selecting  recruits  for  training  in  AFS  3E0X2 
(Electrical  Power  Production),  and  2W1X1F  (F-16  Armament  Systems).  The  ASVAB 
general  aptitude  test  (GEN)  was  a  statistically  significant  predictor  of  success  in  2W0X1 A 
(Munitions  Combat  Ammunition  Systems)  and  3E8X1  (Explosive  Ordinance  Disposal).  In 
addition  the  ASVAB  administrative  aptitude  test  (ADMIN)  was  a  significant  predictor  of 
success  in  AFS  2T2X1  (Air  Transportation  Equipment  Maintenance).  Details  of  these  and 
the  other  logistic  regression  analyses  can  be  found  in  Appendix  C. 

DISCUSSION 

Results  provide  evidence  that  the  recruits  entering  maintenance  technical  training 
courses  between  January,  1990  and  September,  1995  were  high  quality  accessions. 

Despite  concerns  about  declining  MECH  scores  in  the  pool  of  military  service-eligible 
young  people  (Chapman,  1996),  these  recruits  appear  to  have  had  the  requisite  levels  of 
mechanical  aptitude.  The  low  failure  rates  that  occurred  during  this  period  make  it  hard  to 
conclude  otherwise.  Tables  2  and  3  show  that  their  average  MECH  score  was  72.3;  nearly 
all  (99.7  percent)  were  in  the  top  3  enlistment  categories,  and  99.9  percent  had  earned  a 
High  School  Diploma  or  GED. 

The  correlations  in  Table  3  indicate  that  a  linear  relationship  exists  between  MECH 
scores  and  FSGs  for  all  but  two  of  the  AFSs.  However,  finding  significant  correlations 
between  MECH  and  FSG  addresses  only  part  of  the  issue.  The  combination  of  a  low 
failure  rate  and  the  lack  of  FSG  criterion  data  for  students  who  failed  their  technical 
training  courses  limited  the  usefulness  of  linear  regression  and  correlational  techniques  for 
differentiating  between  students  likely  to  complete  training  successfully  and  those  likely  to 
fail. 

Because  of  these  problems,  the  impact  of  lowering  MECH  cut-off  scores  on  training 
course  attrition  could  not  be  estimated  for  all  the  maintenance  occupations  included  in  this 
study.  Furthermore,  the  strength  of  the  evidence  supporting  changes  in  MECH  cut-off 
scores  varied  among  the  maintenance  schools. 

The  contingency  table  (Table  4)  and  logistic  regression  results  (Table  5)  show  that 
MECH  scores  are  only  weakly  related  to  technical  training  failure  for  recruits  selected  for 
maintenance  technical  schools  with  the  present  criteria.  This  does  not  imply  that  MECH 
scores  are  not  important  criteria  for  entry  into  these  occupations.  The  correlations  in  Table 
3  make  it  clear  that  in  all  but  two  cases,  failure  rates  would  be  higher  without  the  use  of 
this  predictor  (Brogden,  1946).  Instead,  it  suggests  that  given  that  a  recruit’s  MECH  score 
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is  high  enough  to  qualify  for  a  maintenance  AFS,  his  or  her  success  in  technical  school 
depends  more  on  other  factors  such  as  motivation,  self-discipline,  interest,  or  disposition 
than  on  the  exact  level  of  his  or  her  MECH  score. 

Recent  studies  (Motowidlo  &  Van  Scotter,  1994;  Van  Scotter  &  Motowidlo,  1996)  have 
shown  that  interpersonal  and  motivational  factors  account  for  a  sizable  proportion  of 
variance  in  the  work  performance  ratings  of  Air  Force  mechanics.  The  same  relationships 
may  hold  true  for  performance  in  training.  These  studies  along  with  other  research  (e.g., 
Barrick  &  Mount,  1991 ;  Tett,  Jackson  &  Rothstein,  1991)  have  shown  that  personality 
traits  such  as  conscientiousness  and  agreeableness  are  important  in  job  performance  in  a 
variety  of  civilian  and  military  occupations. 

RECOMMENDATIONS 


General 

A  selection  or  classification  system  is  only  as  good  as  its  performance  criteria.  Without 
data  on  the  performance  of  both  successful  and  unsuccessful  candidates,  the  selection  and 
classification  system  cannot  support  the  Air  Force’s  war  fighting  mission  effectively. 
Technical  training  written  test  scores  and  trainees’  ratings  on  the  performance  items  used 
in  MRT  hands-on  training  are  essential.  In  the  absence  of  this  data  it  is  impossible  to 
ensure  the  accuracy,  efficiency,  equal  opportunity,  and  fairness  of  personnel  decisions. 
Organizations  using  tests  in  hiring  or  classification  decisions  also  are  ethically  (and 
legally)  responsible  for  ensuring  that  criterion  measures  used  to  validate  those  tests  are 
relevant  and  appropriate  (American  Psychological  Association,  1985). 

Recommendations  for  ensuring  selection  criteria  are  effective. 

1.  Require  schools  to  collect  criterion  data  on  successful  and  unsuccessful  students. 
This  data  should  be  maintained  with  other  personnel  data  files. 

2.  Conduct  research  to  determine  the  reliability  and  validity  of  hands-on  performance 
evaluations. 

3.  Investigate  the  use  of  alternate  (i.e.,  motivational  and  dispositional  predictors  of 
performance). 

Recommendations  on  cut-off  scores. 

4.  For  the  two  courses  listed  below  there  was  no  evidence  of  a  linear  relationship 
between  MECH  scores  and  technical  training  success  (Table  3).  Research  is  needed 
to  develop  and  validate  predictors  for  these  specialties  immediately. 

a.  2E6X1,  Communications  (Antenna) 

b.  3P1X1,  Security  Police  Combat  Arms 
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5.  The  MECH  cut-off  scores  for  the  courses  listed  below  should  be  raised  because  the 
failure  rate  (Table  4)  associated  with  the  current  MECH  cut-off  score  is  unacceptably 
high. 

a.  2A6X2,  Aerospace  Ground  Equipment  (AGE)  Maintenance 

b.  2T3X1,  Special  Purpose  (SP)Vehicle  Maintenance 

c.  2T3X2A,  Fire  Truck  Maintenance 

d.  3E8X1,  Explosive  Ordinance  Disposal 

6.  The  MECH  cut-off  score  for  the  courses  listed  below  should  not  be  lowered  because 
the  failure  rates  for  applicants  with  MECH  scores  below  the  current  cut-off  score  is 
within  one  percent  of  the  five  percent  maximum  (Table  5). 

a.  2A5X1,C-141  Maintenance 

b.  2T3X2B,  Refueling  Vehicle  Maintenance 

7.  For  all  remaining  courses  examined  in  this  study  there  is  no  evidence  that  failure  rates 
would  increase  to  unacceptable  levels  if  the  cut-off  scores  were  gradually  lowered. 

It  is  important  to  remember  that  the  results  reported  here  are  based  on  1990-1995  data. 
Changes  may  have  occurred  in  some  technical  training  courses  after  mid- 1995  that  would 
impact  these  results  and  recommendations. 
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Appendix  A 


Procedures  Used  to  Identify  Criterion  Groups 

The  procedures  used  to  identify  groups  of  trainees  in  an  AFS  who  received  similar 
technical  training,  and  for  whom  similar  criterion  measures  were  available  are  described 
here.  The  goal  was  to  avoid  distorting  the  relationship  between  trainees’  MECH  scores 
and  final  technical  school  grades  (FSG)  by  combining  dissimilar  trainee  groups. 

1 .  Training  course  managers  helped  us  determine  when  significant  changes  had  occurred 
in  any  of  the  AFSs  examined  in  this  study.  Typical  include  changes  include 
implementation  of  Utilization  and  Training  Workshop  recommendations  affecting  course 
content,  sequencing  of  materials,  or  the  length  of  a  course.  During  the  time  period 
encompassed  by  this  study  there  were  also  a  number  of  changes  that  were  out  of  the 
ordinary.  They  were  the  result  of  the  restructuring  of  many  maintenance  career  fields,  the 
closure  of  training  bases  such  as  Chanute  AFB,  as  well  as  changes  that  resulted  from  “Air 
Force  Year  of  Training”  initiatives.  With  the  help  of  the  training  course  managers  1-4 
subgroups  were  identified  for  EACH  AFS.  The  initial  groups  are  listed  in  the  next  few 
pages. 

2.  Mech  scores  and  FSGs  were  computed  for  each  subgroup.  Differences  between  the 
correlations  for  groups  of  trainees  in  the  same  AFS  were  tested  using  using  a  method 
described  by  Cohen  (1988).  Since  the  MECH  test  had  not  been  changed  during  the  period 
covered  by  the  data  used  in  this  study  the  correlations  were  not  expected  to  differ 
significantly  between  groups  of  trainees  in  the  same  AFS  unless  the  course  had  changed  in 
some  important  way.  If  the  correlations  did  differ,  it  seemed  likely  that  it  was  because  of 
changes  in  the  FSG  criterion.  The  test  procedure  was  as  follows: 

a.  Because  the  sample  sizes  («)  differed  among  the  groups  it  was  necessary  to  compute 
the  harmonic  mean  of  the  two  sample  sizes  as  shown  here. 

2(n>  -  3)(m  -  3) 

n  = - - — 

+  m  -  6 

b.  Correlations  were  transformed  into  Fisher  zs  using  tables  provided  by  Cohen  (1988). 

c.  A  test  statistic  was  computed  as  qs= |z1-z2|. 

d.  The  test  statistic  was  compared  the  tabled  criterion  value,  qc  ,  for  ft'  to  determine  if 
the  difference  in  the  rs  was  significant  at  the  p  <.05  level  (Cohen,  1988;  p.139). 

e.  If  the  correlations  differed  significantly  the  samples  were  analyzed  separately. 

If  the  test  did  not  provide  evidence  of  a  difference  the  samples  were  combined  for  the 
remainder  of  the  analyses.  Table  3  shows  the  results  of  this  process. 
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Appendix  C 

Logistic  Regression  Results 

Logistic  regression  analyses  tested  the  usefulness  of  MECH  scores  for  predicting 
success  or  failure  in  training.  For  those  AFSs  for  which  minimum  scores  for  MECH  and 
another  ASVAB  composite  score  are  required,  additional  analyses  using  both  scores  as 
independent  variables  were  conducted. 

Significant  chi-square  statistics  for  the  overall  model  and  the  beta  weight  for  MECH, 
and  non-significant  goodness  of  fit  statistics  provide  evidence  that  MECH  scores  explain 
variance  in  the  pass/fail  criterion.  It  is  also  necessary  to  examine  the  effectiveness  of  the 
model  in  supporting  the  decision-maker's  objective.  The  objective  here  is  to  identify 
trainees  who  are  likely  to  fail.  Since,  none  of  the  models  shown  on  the  next  few  pages 
predicted  any  failures,  the  results  provide  little  support  for  the  use  of  MECH  scores  to 
screen  applicants  for  these  occupations. 
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Appendix  C 

LOGISTIC  REGRESSION  RESULTS  FOR  MECH  SCORE  AS  A  PREDICTOR  OF  TRAINING  SUCCESS/FAILURE 
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